Total field of view classification for head-mounted display

ABSTRACT

Virtual images are located for display in a head-mounted display (HMD) to provide an augment reality view to an HMD wearer. Sensor data may be collected from on-board sensors provided on an HMD. Additionally, other day may be collected from external sources. Based on the collected sensor data and other data, the position and rotation of the HMD wearer&#39;s head relative to the HMD wearer&#39;s body and surrounding environment may be determined. After resolving the HMD wearer&#39;s head position, the HMD wearer&#39;s total field of view (TFOV) may be classified into regions. Virtual images may then be located in the classified TFOV regions to locate the virtual images relative to the HMD wearer&#39;s body and surrounding environment.

BACKGROUND

See-through head-mounted displays (HMDs) provide the ability to augmentwhat the wearer sees with virtual objects. In other words, the HMDaugments the HMD wearer's view of the real world with virtual imagery toprovide an augment reality view. However, those virtual images caneasily become distracting and obtrusive, drawing attention away from thereal-world and occluding the HMD wearer's normal (non-augmented) view.This may present a challenge particularly in applications of “always-on”wearable display devices, especially when considering mobile, outdoor,and in-activity scenarios.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Embodiments of the present invention relate to locating virtual imagesin a head-mounted display (HMD) to provide an augment reality view to anHMD wearer. Sensor data may be collected from on-board sensors providedon an HMD. Additionally, other data may be collected from externalsources. Based on the collected sensor data and other data, the positionand rotation of the HMD wearer's head relative to the HMD wearer's bodyand surrounding environment may be determined. After resolving the HMDwearer's head position, the HMD wearer's total field of view (TFOV) maybe classified into regions. Virtual images may then be located in theclassified TFOV regions to locate the virtual images relative to the HMDwearer's body and surrounding environment.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor use in implementing embodiments of the present invention;

FIG. 2 is a block diagram of an exemplary system for classifying regionsof a TFOV and locating virtual objects in the regions in accordance withan embodiment of the present invention;

FIGS. 3A and 3B provide a profile and front views of an HMD wearer'sTFOV classified with primary, secondary, and tertiary regions inaccordance with an embodiment of the present invention;

FIG. 4 illustrates fixed regions for a HMD wearer's TFOV in accordancewith an embodiment of the present invention;

FIG. 5 illustrates dynamic regions for an HMD wearer's TFOV inaccordance with an embodiment of the present invention;

FIG. 6 illustrates an HMD wearer's view with virtual objects located inprimary, secondary, and tertiary regions in accordance with anembodiment of the present invention;

FIG. 7 is a flow diagram showing a method for classifying regions of anHMD wearer's TFOV in accordance with an embodiment of the presentinvention; and

FIG. 8 is a flow diagram showing a method for displaying virtual objectsusing an HMD in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Embodiments of the present invention are directed to the automatic andcontinuous placement and replacement of virtual images in an HMD'sdisplay relative to the HMD wearer's total field of view (TFOV) toprovide the HMD wearer with an augmented reality view. As used herein,“TFOV” accounts for complete range of rotation and translation of aperson's head. This is in contrast to a person's field of view (FOV),which accounts for the extent of what the person sees at a given moment.

In accordance with embodiments of the present invention, sensor data iscollected from any number of on-board sensors provided on an HMD.Additionally, other data may be collected from sources external to theHMD. The sensor data and other data may be used to track and interpretthe HMD wearer's physical head as it relates the HMD wearer's physicalbody and to the physical environment surrounding the HMD wearer. Bytracking the HMD wearer's physical head relative to the HMD wearer'sbody and surrounding environment, the HMD wearer's TFOV may beclassified into various regions. These regions may be defined relativeto the HMD wearer's body and surrounding environment. In someembodiments, the regions may include a primary region and one or morenon-primary regions. Virtual images may be located within the TFOVaccording to the classified regions. Accordingly, embodiments of thepresent invention provide a system that is aware of an HMD wearer's headrelative to the HMD wearer's body and surrounding environment and mayuse such knowledge to locate virtual images relative to the HMD wearer'sbody and surrounding environment. In some embodiments, the virtualobjects may be generally located away from the primary region and placedin unobtrusive regions such that the virtual images are presented innon-dangerous ways by reducing visual occlusion of the primary region.The virtual objects may also be located away from the primary region inorder to provide context (e.g., classify the information as lessrelevant, less important, etc.).

Accordingly, in one aspect, an embodiment of the present invention isdirected to one or more computer storage media storing computer-useableinstructions that, when used by one or more computing devices, cause theone or more computing device to perform a method. The method includesreceiving sensor data from one or more HMD on-board sensors. The methodalso includes using the sensor data to determine an HMD wearer's headposition and rotation relative to the HMD wearer's body and anenvironment surrounding the HMD wearer. The method further includesclassifying two or more regions within the HMD wearer's TFOV based onone or more pre-determined rules and the HMD wearer's head position androtation relative to the HMD wearer's body and an environmentsurrounding the HMD wearer. The method still further includes locatingvirtual images to be displayed by the HMD based on classifying the twoor more regions within the HMD wearer's TFOV.

In another embodiment, an aspect of the invention is directed to an HMD.The HMD includes one or more on-board sensors. The HMD also includes oneor more processors configured to: determine a position and rotation ofan HMD wearer's head relative to the HMD wearer's body based on sensordata from the one or more on-board sensors, classify two or more regionsof the HMD wearer's TFOV based on the position and rotation of the HMDwearer's head relative to the HMD wearer's body, and locate one or morevirtual objects in the two or more regions. The HMD further includes oneor more display components configured to display at least one of the oneor more virtual objects to provide the HMD wearer an augmented view.

A further embodiment is directed to a method for classifying regions ofan HMD wearer's TFOV. The method include continuously: receiving sensordata from one or more sensors on board the HMD; receiving other datafrom one or more sources external to the HMD; and classifying regions ofthe HMD wearer's TFOV relative to the HMD wearer's body and surroundingenvironment based on the sensor data and other data.

Having briefly described an overview of embodiments of the presentinvention, an exemplary operating environment in which embodiments ofthe present invention may be implemented is described below in order toprovide a general context for various aspects of the present invention.Referring initially to FIG. 1 in particular, an exemplary operatingenvironment for implementing embodiments of the present invention isshown and designated generally as computing device 100. Computing device100 is but one example of a suitable computing environment and is notintended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing device 100be interpreted as having any dependency or requirement relating to anyone or combination of components illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Theinvention may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The invention may alsobe practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

With reference to FIG. 1, computing device 100 includes a bus 110 thatdirectly or indirectly couples the following devices: memory 112, one ormore processors 114, one or more presentation components 116,input/output (I/O) ports 118, input/output components 120, and anillustrative power supply 122. Bus 110 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 1 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component. Also, processors have memory. Theinventors recognize that such is the nature of the art, and reiteratethat the diagram of FIG. 1 is merely illustrative of an exemplarycomputing device that can be used in connection with one or moreembodiments of the present invention. Distinction is not made betweensuch categories as “workstation,” “server,” “laptop,” “hand-helddevice,” etc., as all are contemplated within the scope of FIG. 1 andreference to “computing device.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing device 100. Communication mediatypically embodies computer-readable instructions, data structures,program modules or other data in a modulated data signal such as acarrier wave or other transport mechanism and includes any informationdelivery media. The term “modulated data signal” means a signal that hasone or more of its characteristics set or changed in such a manner as toencode information in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer-readable media.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing device 100includes one or more processors that read data from various entitiessuch as memory 112 or I/O components 120. Presentation component(s) 116present data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

I/O ports 118 allow computing device 100 to be logically coupled toother devices including I/O components 120, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc.

Referring now to FIG. 2, a block diagram is provided that illustrates asystem 200 for classifying an HMD wearer's TFOV and locating virtualobjects in classified regions in accordance with an embodiment of thepresent invention. It should be understood that this and otherarrangements described herein are set forth only as examples. Otherarrangements and elements (e.g., machines, interfaces, functions,orders, and groupings of functions, etc.) can be used in addition to orinstead of those shown, and some elements may be omitted altogether.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed by oneor more entities may be carried out by hardware, firmware, and/orsoftware. For instance, various functions may be carried out by aprocessor executing instructions stored in memory.

As shown in FIG. 2, the system 200 includes an HMD 202. Although notshown, the HMD 202 may include components such as those described abovewith reference to FIG. 1 for the general computing device 100,including, for instance, memory, processors, computer-readable media,input/output components, and a power supply. The HMD 202 may be providedin any of a number of different form factors, including, for instance,glasses, goggles, or a helmet.

The HMD 202 may generally provide an augmented view of the real world toa wearer by augmenting the wearer's view of the real world withcomputer-generated virtual images. To provide the wearer with anaugmented view, the HMD 202 may include a display component 204 thatdisplays computer-generated virtual images while still allowing thewearer to view the real world. In some embodiments, this may includeheads-up display (HUD) technology, which may employ, for instance, anytype of projection or microdisplay technology to provide the virtualimages. Other technologies may also be employed, such as retinaldisplays in which images are projected directly onto the wearer's retinawhile the wearer is viewing the real world.

The HMD 202 may also include a wireless communication component 206 thatprovides wireless communication capabilities to allow the HMD 202 tocommunicate with companion devices (e.g., smartphones), server devices,or other network components. For instance, some embodiments of thepresent invention may be implemented by having a cloud-based serviceassist the HMD 202 in classifying regions of the HMD wearer's TFOV andlocating virtual objects in those regions. In some embodiments, the HMD202 may be configured to communicate directly with server devices; whilein other embodiments, the HMD 202 may communicate with server devicesvia a companion device local to the HMD wearer (e.g., a smartphone orother device).

The HMD 202 may also include a number of on-board sensors 208 thatprovide positional and other sensor data. Any of a variety of differenttypes of on-board sensors 208 may be included on the HMD 202 inaccordance with various embodiments of the present invention. Generally,any sensor that allows the system to determine the HMD wearer's headposition and rotation relative to the HMD wearer's body and surroundingenvironment, eye position, or other useful situational information maybe employed. By way of example only and not limitation, the on-boardsensors 208 may include GPS sensors, inertial measurement unit (IMU)sensors, depth sensors, cameras, eye tracking sensors, microphones,biometric sensors, and other types of sensors.

An IMU on the HMD 202 may measure inertial acceleration and mayincorporate the functionality of accelerometers, gyroscopes,magnetometer, and other devices. In one embodiment, an IMU measurementof inertial acceleration has six degrees of freedom. Data from an IMUmay provide relative movement and rotation information that may beemployed with other data to infer absolute location in space. Theadditional data may include information from GPS sensors, which may beemployed to provide location information on a macro level. In someembodiments, the HMD 202 may employ assisted GPS (A-GPS).

Depth sensors and cameras may be employed on the HMD 202 to collect dataregarding the environment surrounding the HMD wearer. Depth sensors maygenerally include sensors that determine the distance to objects. Insome implementations, a depth sensor may include an infrared sensor thatcaptures IR light emitted from a transmission source and reflected froman object. Distance data may be determined using time-of-flight,triangulation, or other known principles. Cameras may be IR or visiblespectrum, black and white or red-green-blue (RGB) cameras. In someembodiments, the parallax between images from two different cameras canbe used to measure depth much as a person's two eyes provide depthperception.

The HMD 202 may also include one or more eye-tracking sensors, whichtypically track the movement of the pupil or other portions of the eyeor the area around the eye to determine the direction of a user's gaze.This may be accomplished, for example, using IR or RGB cameras aimed atthe HMD wearer's eyes. Microphones may be provided to collect audioinformation, including the HMD wearer's voice and surroundingenvironmental audio. Biometric sensors may be used to collect biometricinformation for the HMD wearer.

Sensor data from the on-board sensors 208 may be provided to a virtualimaging system 210 for processing. In some embodiments of the presentinvention, the virtual imaging system 210 may be provided on the HMD202. In other embodiments, the virtual imaging system 210 may beprovided by a device external to the HMD 202, such as a server device orother network component. In still further embodiments, the virtualimaging system 210 may be distributed across multiple devices (e.g., theHMD 202 and an external device). Any and all such variations arecontemplated to be within the scope of embodiments of the presentinvention.

The virtual imaging system 210 includes a head position component 212that is operable to receive sensor data from the on-board sensors 208and determine the HMD wearer's head position and rotation relative toHMD wearer's surrounding environment and the HMD wearer's body. By wayof example only and not limitation, the head position component 212 mayemploy techniques such as simultaneous location and mapping (SLAM)using, for instance, camera data and/or depth sensor data to provide areal-time position of the HMD wearer's head relative to a mappedsurrounding environment. IMU data may also provide relative rotation andposition information even when the camera or depth sensors areunavailable. Regions of the HMD wearer's body (e.g., hand, arm, torso,legs, feet, etc.) may also be identified using sensor data. Forinstance, camera data and/or depth sensor data may be obtained when theHMD wearer is looking at him or herself. As such, the HMD wearer's headposition and rotation relative to the HMD wearer's body may be inferredto a useful degree of accuracy. This may include information such aswhether the HMD wearer is standing, sitting, looking straight aheadrelative to torso, to name a few.

The head position component 212 may also receive data from sensors andother sources 220 external to the HMD 212. By way of example only andnot limitation, the other sources 220 may include external cameras,external depth sensors (e.g., a KINECT sensor), other HMDs, mobiledevices, and historical sensor data stored remotely from the HMD.Generally, any external source of information that allows the system todetermine the HMD wearer's head position and rotation relative to theHMD wearer's body and surrounding environment, eye position, or otheruseful situational information may be employed.

Data from the various on-board sensors 208 and external sources 220 maygenerally provide redundancies and refinement. However, it should beunderstood that not all sensors are necessary. Any combination ofsensors indicated herein, as well as other sensors and sources ofinformation, may be employed within the scope of embodiments of thepresent invention.

After the head position component 212 resolves the HMD wearer's headposition and rotation relative to the HMD wearer's body and surroundingenvironment, the TFOV classification component 214 classifies regions ofthe HMD wearer's TFOV. In accordance with embodiments of the presentinvention, the TFOV classification component 214 identifies regions ofthe HMD wearer's TFOV according to various rules. The rules may beuser-defined or system-defined. Additionally, the rules may beimmutable, dynamic based on different inputs regarding the HMD wearer'scurrent situation and environment, or alterable by the HMD wearer toprovide the best experience for the HMD wearer based on what the HMDwearer is currently doing.

In some embodiments of the present invention, the regions may include aprimary region and any number of additional non-primary regions. Theprimary region may correspond with a primary area within the HMDwearer's FOV. Generally, this corresponds with an area that should notbe occluded with virtual objects. The non-primary regions correspondwith areas that may be occluded with virtual objects.

By way of example, FIGS. 3A and 3B illustrate a profile view and frontview, respectively, of a TFOV 302 for an HMD wearer 304. As shown inFIGS. 3A and 3B, the TFOV 302 has been classified into a primary region306, a secondary region 308, and a tertiary region 310. It should benoted that the classification shown in FIGS. 3A and 3B are provided asexamples only, and the TFOV may generally be classified into any numberof regions. The regions may be symmetrical or non-symmetrical.Additionally, the regions may be divided vertically, horizontally, orany permutation thereof. In some embodiments, the regions may beclassified relative to the HMD wearer's body or body parts. Forinstance, the regions may comprise an upper left of body region, upperright of body region, lower left region of body, and lower right regionof body.

The classification of regions of the TFOV may be fixed or maydynamically change. By way of illustration, examples of fixed regionsversus dynamic regions are provided in FIGS. 4 and 5. Initially, FIG. 4illustrates an example of fixed regions. As shown in FIG. 4, as the HMDwearer moves his/her head looking up and down, the regions remain fixed.In contrast, FIG. 5 illustrates an example of dynamic regions. As shownin FIG. 5, as the HMD wearer moves his/her head up and down, the regionsmove with the HMD wearer's head movements.

TFOV regions may dynamically change by automatically adjusting the size(i.e., expanding or shrinking) of regions, shifting regions within theTFOV, or completely re-classifying new regions for the TFOV. The regionsmay automatically change based on rules triggered by events,environmental conditions, body placement and movement, and additionalsensor information. For instance, if the HMD wearer is sitting, theprimary region may be relatively small since the HMD wearer isstationary and it's safer to occlude more of the HMD wearer's FOV.However, if the HMD wearer stands and begins to walk, the primary regionmay expand such that less of the HMD wearer's FOV is occluded withvirtual images. If the HMD wearer begins running or begins to drive acar, the primary region may expand even further. As an example ofre-classifying regions, suppose the HMD wearer's eyes are gazing andhands are moving in a secondary region of the TFOV. This may beindicative that the HMD wearer is performing some task in that space. Asa result of these inputs, the TFOV may be re-classified such that regionis now the primary region. Other environmental conditions may alsoaffect the classification of regions. For instance, biometric inputs mayindicate that the HMD wearer is nervous or frightened. In response, theprimary region may expand.

In accordance with some embodiments of the present invention,combinations of fixed regions and dynamically changing regions may beemployed over time by employing rules that control whether fixed regionsor dynamic regions are employed at any given time. In particular, somecircumstances may dictate the use of fixed regions, while othercircumstances may dictate the use of dynamic regions. For instance, theregions may initially be fixed for a given time period. When certaininputs are received, rules may then dictate dynamically changing theregions. By way of specific example, if the HMD wearer is sitting, theregions may remain fixed while the HMD wearer remains sitting. However,once the HMD wearer gets up, rules may be triggered to cause the regionsto dynamically change as the user moves around in space. Accordingly,the region classifications may be environmentally-adaptive.

After classifying regions of the TFOV, the virtual object placementcomponent 216 may locate virtual objects in the various regionsaccording to the classifications. Generally, the virtual objects may beplaced in regions according to defined rules. The rules may beuser-defined or system-defined. Additionally, the rules may beimmutable, dynamic based on different inputs regarding the HMD wearer'scurrent situation and environment, or alterable by the HMD wearer toprovide the best experience for the HMD wearer based on what the HMDwearer is currently doing. In some embodiments, this may include placingmost virtual objects into non-primary regions of the TFOV to preventoccluding the primary region. In some instances, only notifications,less intrusive virtual objects, or virtual objects deemed to be moreimportant may be located in the primary region.

By way of illustration, FIG. 6 provides a view provided to an HMD wearerin accordance with an embodiment of the present invention. As shown inFIG. 6, the HMD wearer is currently viewing another person 602. The HMDwearer's FOV covers a primary region 604, secondary region 606, andtertiary region 608 of the TFOV. It should be noted that the dashedlines and region labels are provided in FIG. 6 for discussion purposes,and those would not be displayed to the HMD wearer. Instead, only thevirtual objects 610 would be displayed. As shown in FIG. 6, only smallernotification virtual objects 610 are displayed within the primary region604, while larger, more intrusive virtual objects 610 are displayedwithin the secondary region 606 and tertiary region 608. As a result,the primary region 604 remains generally unobstructed by virtualobjects.

In various embodiments of the present invention, a permission set may beemployed to govern placement of content within the TFOV. This mayinclude universal rules that span across applications or circumstancesthat dictate to place or not place specific content in specific regions.There may be also be changing TFOV permissions based on status (e.g.,rules for placement of virtual objects when the user is driving may bedifferent from rules for placement of virtual objects when the user isin a living room). There may also be permission sets that change basedon the classification of the application being run by the system. Theinterplay between these global/state/application-based permissionsdrives the content and the location of the content displayed within theTFOV.

In embodiments in which regions are classified relative to the HMDwearer's body, virtual objects may be located in the regions in order tolocate those virtual objects relative to the HMD wearer's body. Forinstance, a region may be classified around the HMD wearer's foot as afoot region, and a particular virtual object may be located in thatregion in order to located the virtual object near the HMD wearer'sfoot. As another example, a region surrounding the HMD wearer's torsomay be classified as a torso region, and a virtual desktop may belocated in that region to contain information the HMD wearer may want toaccess. As such, the HMD wearer may view the information by looking downat the virtual desktop surrounding the HMD wearer's torso.

In some instances, the virtual objects may include static objects thatare not interactive. For instance, virtual objects may simply displayinformation or graphics for the HMD wearer to view. In other instances,the virtual objects may be user interface (UI) objects with which theHMD wearer may interact, using gestures, voice commands, and/or otherforms of input.

Because the TFOV extends beyond the HMD wearer's FOV, only a portion ofall virtual objects within the TFOV may be viewed by the HMD wearer atany give time. Accordingly, the virtual image display component 218 maybe operable to determine the HMD wearer's current FOV within the TFOVand to cause virtual objects to be displayed in the appropriatelocations within the HMD wearer's FOV. For instance, if a virtual objectis located in a TFOV region near the HMD wearer's feet, the virtualobject would not be displayed when the HMD wearer is looking up.However, if the HMD wearer were to look down, the virtual image displaycomponent 218 would determine that the HMD wearer's FOV is over the footregion and would cause the virtual object to be displayed at theappropriate location. It should be noted that in some embodiments, theextent of the HMD wearer's FOV may simply be considered to correspondwith the extent of the display area of the display component 204 of theHMD 202.

In some embodiments, the HMD's FOV may extend beyond the HMD wearer'sFOV. For instance, the depth sensors, cameras, and other on-board HMDsensors, as well as external sensors may be able to capture data outsideof the HMD wearer's FOV. As such, the HMD's FOV may be defined by theextent to which the various sensors cover, which may extend well beyondthe HMD wearer's FOV. Some embodiments may take advantage of the HMD'sFOV by allowing an HMD wearer to interact with a virtual object that isnot currently displayed to the HMD wearer because it is outside the HMDwearer's FOV, as long as the virtual object is located within the HMD'sFOV. For instance, suppose an interactive virtual object is locatedadjacent in a region adjacent to the HMD wearer's right hip. The HMDwearer may interact with the virtual object even when the HMD wearer islooking elsewhere and the virtual object is not being displayed to theHMD wearer, as long as the area adjacent to the HMD wearer's right hipat which the virtual object is located is within the HMD's FOV such thatthe HMD may capture the HMD wearer's interaction with the virtualobject.

Referring now to FIG. 7, a flow diagram is provided that illustrates amethod 700 for classifying regions of an HMD wearer's TFOV in accordancewith an embodiment of the present invention. The method 700 may beperformed, for instance, by an HMD, a device external to the HMD, or acombination thereof. As shown at block 702, data from on-board sensorsprovided on an HMD is received. As discussed previously, the on-boardsensors may include, without limitation, GPS sensors, inertialmeasurement unit (IMU) sensors, depth sensors, cameras, eye trackingsensors, microphones, biometric inputs, and other sensors. Additionally,data may be received from sources external to the HMD, as shown at block704. By way of example only and not limitation, this may includeexternal cameras, external depth sensors, other HMDs, mobile devices,and historical sensor data stored remotely from the HMD.

Based on the sensor data from the on-board sensors and other data fromthe external sources, the HMD wearer's head position and rotation aredetermined relative to the HMD wearer's body and surroundingenvironment, as shown at block 706. Using that information, the HMDwearer's TFOV is classified into two or more regions, as shown at block708. In accordance with embodiments of the present invention, the TFOVis classified according to rules that may be user-defined orsystem-defined and immutable or dynamic. Virtual objects are located inthe regions of the TFOV, as shown at block 710. A number of rules mayalso be defined for placing the various virtual objects into theregions. These rules may also be user-defined or system-defined andimmutable or dynamic. In some embodiments, the system may take intoaccount the applications that are being accessed when determining whereto locate the various virtual objects. As represented by the return toblocks 702 and 704, the process of classifying regions of the TFOV maybe repeated. In some embodiments, one or more rules may triggerreclassifying regions of the TFOV, while in other embodiment, the TFOVmay be continuously reclassified.

Turning to FIG. 8, a flow diagram is provided that illustrates a method800 for displaying virtual objects using an HMD in accordance with anembodiment of the present invention. As shown at block 802, the HMDwearer's FOV is initially determined. In some embodiments, the extent ofthe HMD wearer's FOV may simply correspond with the extent the displayarea provided by a display component of the HMD. As shown at block 804,the HMD wearer's FOV is compared with regions of a TFOV that have beendetermined, for instance, using the method 700 described above withreference to FIG. 7. The comparison identifies the placement of the HMDwearer's FOV relative to the regions of the TFOV.

The virtual objects to display are determined at block 806 based on thelocation of the HMD wearer's FOV relative to the TFOV regions and thelocation of virtual objects in the TFOV regions. In some embodiments, aset of permissions may be applied at this point to govern usage ofvirtual objects based on, for instance, the current user state or thespecific application(s) being engaged. The virtual objects that areidentified are then displayed using a display component of the HMD atthe appropriate location, as shown at block 808.

As can be understood, embodiments of the present invention provide forclassifying regions of an HMD wearer's TFOV based on sensor inputs andrules and locating virtual objects in the classified regions. Thepresent invention has been described in relation to particularembodiments, which are intended in all respects to be illustrativerather than restrictive. Alternative embodiments will become apparent tothose of ordinary skill in the art to which the present inventionpertains without departing from its scope.

From the foregoing, it will be seen that this invention is one welladapted to attain all the ends and objects set forth above, togetherwith other advantages which are obvious and inherent to the system andmethod. It will be understood that certain features and subcombinationsare of utility and may be employed without reference to other featuresand subcombinations. This is contemplated by and is within the scope ofthe claims.

1. One or more computer storage media storing computer-useableinstructions that, when used by one or more computing devices, cause theone or more computing device to perform a method, the method comprising:receiving sensor data from one or more head-mounted display (HMD)on-board sensors; using the sensor data to determine an HMD wearer'shead position and rotation relative to the HMD wearer's body and anenvironment surrounding the HMD wearer; classifying two or more regionswithin the HMD wearer's total field of view (TFOV) based on one or morepre-determined rules and the HMD wearer's head position and rotationrelative to the HMD wearer's body and an environment surrounding the HMDwearer; and locating virtual images to be displayed by the HMD based onclassifying the two or more regions within the HMD wearer's TFOV.
 2. Theone or more computer storage media of claim 1, wherein the one or moreHMD on-board sensors comprise one or more selected from the following: aGPS sensor, an inertial measurement unit sensor, a depth sensor, acamera, an eye tracking sensor, a microphone, and a biometric sensor. 3.The one or more computer storage media of claim 1, wherein the methodfurther comprises: receiving other data from one or more sourcesexternal to the HMD; and using the other data in conjunction with thesensor data to determine the HMD wearer's head position and rotationrelative to the HMD wearer's body and the environment surrounding theHMD wearer.
 4. The one or more computer storage media of claim 3,wherein the one or more sources external to the HMD comprise an externalcamera, an external depth sensor, another HMD, a mobile device, andhistorical sensor data stored remotely from the HMD.
 5. The one or morecomputer storage media of claim 1, wherein the two or more regions arefixed within the HMD wearer's TFOV.
 6. The one or more computer storagemedia of claim 1, wherein the two or more regions dynamically changewithin the HMD wearer's TFOV.
 7. The one or more computer storage mediaof claim 1, wherein the two or more regions include a primary region andone or more non-primary regions.
 8. The one or more computer storagemedia of claim 7, locating the virtual images comprises locating thevirtual images to reduce occluding the primary region.
 9. The one ormore computer storage media of claim 1, wherein classifying the two ormore regions within the HMD wearer's TFOV comprises classifying at leastone of the regions relative to the HMD wearer's body.
 10. The one ormore computer storage media of claim 9, wherein locating the virtualreality images to be displayed by the HMD comprises locating at leastone of the virtual reality images relative to HMD wearer's body based onclassifying the at least one of the regions relative to the HMD wearer'sbody.
 11. The one or more computer storage media of claim 1, wherein themethod further comprises: determining a field of view (FOV) for the HMDwearer; comparing the FOV with the two or more regions of the HMDwearer's TFOV; determining a subset of virtual objects to display basedon a location of the FOV relative to the two or more regions of the HMDwearer's TFOV; and displaying the subset of virtual objects via adisplay component of the HMD.
 12. The one or more computer storage mediaof claim 1, wherein the virtual images comprise at least one staticvirtual image and at least one interactive virtual image.
 13. The one ormore computer storage media of claim 1, wherein the method furthercomprises identifying user interaction with an interactive virtual imagelocated in a first region from the two or more regions, the first regionbeing outside the HMD wearer's current field of view, and performing anaction in response to the user interaction with the interactive virtualimage.
 14. A head-mounted display (HMD) comprising: one or more on-boardsensors; one or more processors configured to: determine a position androtation of an HMD wearer's head relative to the HMD wearer's body basedon sensor data from the one or more on-board sensors, classify two ormore regions of the HMD wearer's TFOV based on the position and rotationof the HMD wearer's head relative to the HMD wearer's body, and locateone or more virtual objects in the two or more regions; and one or moredisplay components configured to display at least one of the one or morevirtual objects to provide the HMD wearer an augmented view.
 15. The HMDof claim 14, wherein the HMD further comprises a wireless communicationcomponent.
 16. The HMD of claim 14, wherein the wireless communicationcomponent is configured to receive other data from one or more sourcesexternal to the HMD, and wherein the one or more processors are furtherconfigured to determine the position and rotation of an HMD wearer'shead relative to the HMD wearer's body based on the other data from theone or more sources external to the HMD.
 17. A method for classifyingregions of an head-mounted display (HMD) wearer's total field of view(TFOV), the method comprising: continuously: receiving sensor data fromone or more sensors on board the HMD; receiving other data from one ormore sources external to the HMD; and classifying regions of the HMDwearer's TFOV relative to the HMD wearer's body and surroundingenvironment based on the sensor data and other data.
 18. The method ofclaim 17, wherein the regions include a primary region and one or morenon-primary regions.
 19. The method of claim 18, wherein the methodfurther comprises locating virtual images in the regions based onclassifying the regions.
 20. The method of claim 19, wherein locatingthe virtual images comprises locating the virtual images in the regionsto reduce occluding the primary region.